Skip to content

feat(prose-tests): judge the walk as told, not the summary returned - #561

Merged
leeovery merged 1 commit into
mainfrom
fix/prose-tests-capture-the-walk
Jul 27, 2026
Merged

feat(prose-tests): judge the walk as told, not the summary returned#561
leeovery merged 1 commit into
mainfrom
fix/prose-tests-capture-the-walk

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • We were judging one message and calling it a walk. An agent returns a single final message; a walk runs across dozens of turns. Every step the walker narrated at the time and then compressed out of its closing summary read to the asserter as a step never taken.
  • This was the under-reporting we spent a long time attributing to the walker, then to the model. It was neither. A Sonnet walk of continue-feature was marked down for skipping the selection menu — its transcript shows it emitted the display, emitted the menu verbatim, stopped, consumed the scripted answer, and validated the selection. An Opus walk of the same case made byte-identical tool calls.
  • The runtime already writes every turn, and the stop payload names the file. The hook now lifts those turns into the world beside the action log. Nothing is asked of the walker — the same property that makes the action log trustworthy.
  • So the walker is asked for less. Its final message no longer has to be a complete log (the instruction that caused the compression). Commands, outputs and file writes leave its list entirely — the harness records those with more fidelity. What remains is only what no tool call can show: the arm entered, the guard that selected it, what went on screen.
  • The asserter drops "a thin narrative is not a missing step" — it papered over this blindness and is now false.
  • A missing walk log is refused loudly, like a missing action log, rather than silently falling back to the closing summary.

Test plan

  • node --test across recorder, corpus and snapshot suites — 40/40 (two new cases: turn ordering and preservation of mid-walk emissions; empty-transcript no-op)
  • Conventions lint clean
  • Verified against a live walk: the asserter prompt now contains Step 0: Initialisation, 0.1 Casing Conventions, 0.2 Boot with its Otherwise arm, and 0.3 Knowledge Gate — none of which appeared in the message the walker returned

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561 👈 current
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575
  32. feat(prose-tests): conduct — playing a person where a script cannot reach #576
  33. fix(prose-tests): what a write put in a file is evidence, not incident #577
  34. fix(investigation): the interview starts from the carrier, not over it #578
  35. test(prose): quick-fix scoping, the whole definition stage in one walk #579
  36. fix(prose-tests): the world log is the walker's record #580
  37. fix(prose-tests): a confirmed failure reports both runs' checks #581
  38. fix(prose-tests): each prescribed command runs as written #582
  39. fix(scoping): context gathering reads the whole carrier, asks only the gaps #583

This was referenced Jul 26, 2026
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from 7fbe33a to 381b839 Compare July 27, 2026 14:37
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from 8659739 to fa64910 Compare July 27, 2026 14:39
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from 381b839 to 02a8b94 Compare July 27, 2026 14:39
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from fa64910 to 50b99a4 Compare July 27, 2026 14:41
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from 02a8b94 to c54eac7 Compare July 27, 2026 14:41
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from 50b99a4 to a585648 Compare July 27, 2026 14:43
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from c54eac7 to 846f22d Compare July 27, 2026 14:43
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from a585648 to 38acfa0 Compare July 27, 2026 14:45
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from 846f22d to fa1572b Compare July 27, 2026 14:45
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from 38acfa0 to c87d156 Compare July 27, 2026 14:47
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from fa1572b to f8d7e6a Compare July 27, 2026 14:47
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from c87d156 to ada9b7a Compare July 27, 2026 14:49
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from f8d7e6a to aabd85e Compare July 27, 2026 14:49
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from ada9b7a to 5a3fb58 Compare July 27, 2026 14:51
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from aabd85e to be0c720 Compare July 27, 2026 14:51
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from 5a3fb58 to 62fbf86 Compare July 27, 2026 15:42
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from be0c720 to b77aa57 Compare July 27, 2026 15:42
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from 62fbf86 to a6da773 Compare July 27, 2026 15:44
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from b77aa57 to a7b1fa8 Compare July 27, 2026 15:44
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from a6da773 to 62bef70 Compare July 27, 2026 15:45
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from a7b1fa8 to a28b049 Compare July 27, 2026 15:45
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from 62bef70 to e68fd0a Compare July 27, 2026 15:47
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from a28b049 to b9a3d00 Compare July 27, 2026 15:47
@leeovery
leeovery force-pushed the fix/prose-tests-record-command-output branch from e68fd0a to 749f002 Compare July 27, 2026 15:49
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from b9a3d00 to 826176b Compare July 27, 2026 15:49
@leeovery
leeovery changed the base branch from fix/prose-tests-record-command-output to main July 27, 2026 15:50
An agent returns one message. A walk happens across dozens of turns. We
were reading only the return value and judging it as though it were the
whole walk — so every step the walker narrated at the time and then
compressed out of its closing summary read as a step never taken.

This is what produced the under-reporting we spent a long time blaming on
the walker, and then on the model. It was neither. A Sonnet walk of
continue-feature was marked down for skipping the selection menu; its
transcript shows it emitted the display, emitted the menu verbatim,
stopped, consumed the scripted answer and validated the selection. An
Opus walk of the same case made byte-identical tool calls. The evidence
was there the whole time and we were discarding it.

The runtime already writes every turn to a transcript, and the stop
payload names its path. The hook now lifts the turns into the world
beside the action log, where the judging happens. Nothing is asked of the
walker, which is what makes it dependable — the same reason the action
log is trusted.

That in turn lets the walker be asked for less. It no longer has to make
its final message a complete log, which was the instruction driving the
compression; commands, outputs and file writes come off its list
entirely, since the harness records those with more fidelity than a
narrative can. What remains is only what no tool call can show: the arm
entered, the guard that selected it, and what went on screen.

The asserter drops the caveat that a thin narrative is not a missing
step. It existed to paper over this blindness and is now false — the walk
it receives is complete, so an absent step is absent.

Missing walk data is a harness failure, refused loudly like a missing
action log, never quietly downgraded to judging the closing summary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the fix/prose-tests-capture-the-walk branch from 826176b to a921fb8 Compare July 27, 2026 15:50
@leeovery
leeovery merged commit b82e05b into main Jul 27, 2026
@leeovery
leeovery deleted the fix/prose-tests-capture-the-walk branch July 27, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant